Maximizing value of genetic sequence data requires an enabling environment and urgency.

Severe price spikes of the major grain commodities and rapid expansion of cultivated area in the past two decades are symptoms of a severely stressed global food supply. Scientific discovery and improved agricultural productivity are needed and are enabled by unencumbered access to, and use of, genetic sequence data. In the same way the world witnessed rapid development of vaccines for COVID-19, genetic sequence data afford enormous opportunities to improve crop production. In addition to an enabling regulatory environment that allowed for the sharing of genetic sequence data, robust funding fostered the rapid development of coronavirus diagnostics and COVID-19 vaccines. A similar level of commitment, collaboration, and cooperation is needed for agriculture.

Enabling machine learning-ready HPC ensembles with Merlin

With the growing complexity of computational and experimental facilities, many scientific researchers are turning to machine learning (ML) techniques to analyze large scale ensemble data. With complexities such as multi-component workflows, heterogeneous machine architectures, parallel file systems, and batch scheduling, care must be taken to facilitate this analysis in a high performance computing (HPC) environment. In this paper, we present Merlin, a workflow framework to enable large ML-friendly ensembles of scientific HPC simulations. By augmenting traditional HPC with distributed compute technologies, Merlin aims to lower the barrier for scientific subject matter experts to incorporate ML into their analysis. As a producer–consumer workflow model, Merlin enables multi-machine, cross-batch job, dynamically allocated yet persistent workflows capable of utilizing surge-compute resources. Key features of Merlin are a flexible HPC-centric interface, low per-task overhead, multi-tiered fault recovery, and a hierarchical sampling algorithm that allows for O(N) task execution and O(NlnN) task queuing to ensembles of millions of tasks. In addition to Merlin’s design, we test the algorithm’s performance in an HPC center and demonstrate the ability to enqueue 40 million simulations in 100 s, with a 30 millisecond per-task overhead that is independent of ensemble size. Finally, we describe some example applications that Merlin has enabled on leadership-class HPC resources, such as the ML-augmented optimization of nuclear fusion experiments and the calibration of infectious disease models to study the progression of and possible mitigation strategies for COVID-19.